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Introduction 

This  paper  looks  forward  from  the  last  Nato  review 
of  computer-based  assessment  (CBA)  of  military 
personnel  (Burke  and  Van  Raay,  1993;  see  also 
Burke,  1993,  and  Burke  et  al.,  1995).  At  the  time 
of  that  report,  research  and  development  among 
Nato  nations  could  be  summarised  according  to 
three  areas  of  work: 

• Desktop  systems  delivering  traditional  tests 
and  questionnaires  as  well  as  more  dynamic 
tasks  developed  from  paradigms  from 
cognitive  psychology,  using  LANs  and  WANs, 
and  from  which  the  principle  gains  were  the 
increased  reliability  and  reduced  costs  from 
automation  of  the  assessment  process.  Systems 
characterising  this  approach  included  those 
developed  by  the  Royal  Air  Force  (RAF)  in  the 
UK  for  officer  and  aircrew  selection 
(Burke,  1992,  and  Burke  et  al.,  1994),  Project 
A in  the  US,  Taskomat  in  the  Netherlands  and 
the  ESPACE  system  in  France. 

• Simulation-based  assessment  (SBA)  systems 
for  delivery  of  sophisticated  work  sample 
measures  usually  administered  after  prior 
screening  using  paper-and-pencil  or  desktop 
tests  and  questionnaires,  and  developed  for 
selection  to  high  risk/high  cost  roles  such  as 
aircraft  pilot.  Systems  characterising  this 
approach  included  the  CAPSS  system  in 
Canada  and  the  GUTS  in  Belgium,  as  well  as  a 
range  of  systems  developed  in  Germany. 
The  primary  focus  of  these  systems  was 
increased  validity  and  reduced  training  costs 
against  which  the  substantial  costs  of  SBA 
development  and  administration  could  be 
recovered. 

• Adaptive  testing  systems  that  sought  to  exploit 
capabilities  unique  to  CBA  in  delivering 
tailored  testing  (i.e.  measurement  geared  to  an 
individual’s  level  of  ability).  The  US 
CAT-ASVAB  programme  stands  as  the  most 


substantial  work  in  this  area  to  date.  As  well  as 
adaptive  testing,  the  late  1980s  and  early  1990s 
also  saw  the  advent  of  item  generation 
techniques  in  which  item  engines  contained  in 
the  test  software  produce  the  item  or  task 
on-the-fly  during  a test  administration.  The  UK 
British  Army  Recruit  Battery  (BARB)  system 
was  the  first  item  generative  system  to  go  live 
in  military  assessment,  though  the  same 
methodology  was  also  used  to  produce  fixed 
parallel  forms  of  paper-and-pencil  tests  for  the 
Royal  Navy  (the  ABC  test  battery). 

The  driver  for  the  development  of  these  systems 
was  the  expectation  of  gains  in  the  utility  of 
military  selection  processes.  Utility  is  defined  as 
the  financial  gain  from  a selection  process  and, 
as  shown  in  Equation  1,  is  dependent  on  a 
number  of  factors: 

• The  validity  of  the  process  in  predicting  the 
outcome  of  interest  (e.g.  training  success,  job 
performance,  turnover)  as  represented  by  rxy 
in  Equation  1 . 

• The  selection  ratio  or  average  quality  of 
those  selected  as  shown  in  the  equation 
by  Zx- 

• The  variability  in  performance  as 
represented  by  SDy. 

• The  numbers  selected  or  N. 

• The  cost  of  the  process  as  represented  by  C. 

U = {Zx.rxy.SDs.N)-C 


Equation  1:  Utility  of  a selection  process 

The  driver  of  greater  utility  still  stands  and,  while 
specific  numbers  will  not  be  quoted,  what  follows 
in  this  paper  is  the  outline  of  a process  that  seeks  to 
maximise  U by  ensuring  the  attraction  of 
applicants,  the  sifting  of  those  applicants  through 
an  integrated  framework,  and  the  management  and 
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harvesting  of  data  through  the  backbone  offered  by 
the  model  suggested. 

Technology  today  six  years  on 

Bartram  (1999)  has  recently  provided  a summary 
of  developments  in  the  technologies  underpinning 
CBA  that  serves  to  update  the  field  of  play  since 
1993.  Among  the  most  significant  developments 
are: 

• Widespread  availability  of  inexpensive  high- 
powered  computers  either  at  work,  home  or 
places  of  learning  (e.g.  colleges,  universities, 
libraries). 

• Greater  reliability  in  software  and  hardware, 
critical  to  CBA  systems. 

• Improved  user  interfaces  and  facilities  such  as 
sound  and  video  cards. 

• Greater  connectivity  with  the  advent  of  the 
internet  and  e-commerce. 

• Greater  convergence  towards  common 
standards  for  hardware  and  software,  which 
has  brought  with  it  the  capability  to  deliver 
more  sophisticated  applications  over  networks 
and  the  internet. 

To  these  can  be  added  ever  increasing  connectivity 
such  as  surfing  the  web  from  home  television  sets 
(or,  for  that  matter,  any  place  in  which  a person  is 
sedentary  for  any  period  such  as  their  car),  and 
ever  increasing  transportability  as  offered  by 
access  to  the  internet  or  company  intranets 
(or,  indeed,  extranets)  via  mobile  phone  and 
palmtop  technology,  technologies  by  the  way 
encouraged  by  military  requirements  such  as 
battlefield  communications. 

It  is  also  worth  mentioning  some  of  the  statistics 
now  being  quoted  for  the  internet  to  give  a sense  of 
the  real  impact  this  is  having  on  the  recruitment 
and  selection  market  (again,  I am  grateful  to  Dave 
Bartram  for  these  statistics): 

• There  are  currently  estimated  to  be  130  million 
internet  users  world  wide,  and  this  is  expected 
to  double  in  the  next  five  years. 

• By  2002,  predictions  suggest  that  98%  of  large 
companies,  85%  of  medium  sized  and  45%  of 
small  companies  will  be  online. 

In  relation  to  the  recruitment  market,  the  launch  of 
the  LAI  Compass  site  by  SHL  in  the  USA  resulted 
in  40,000  hits  within  the  first  few  days,  far  greater 


than  expected.  In  the  UK,  the  launch  of  the 
Kom/Ferry  site  drew  10,000  hits  within  the  first  six 
weeks.  A recent  Institute  of  Personnel 
Development  report  issued  in  the  UK  shows  that 
the  use  of  internet  recruitment  had  grown  to  32% 
of  UK  companies  by  May  of  this  year  (IPD,  1999). 
The  most  frequently  visited  sites  in  the  US  are 
those  with  the  word  career  in  their  URLs. 

Today,  then,  the  biggest  contrast  with  what  was 
CBA  state-of-the-art  in  1993  is  the  advent  of 
distributed  assessment  and  this  will  be  the  focus  of 
the  model  described  next. 

A model  for  a distributed  assessment  system 

Figure  1 below  shows  a template  for  a process 
supported  and  delivered  by  a Distributed 
Assessment  System  or  DAS.  What  this  template 
shows  is,  in  fact,  a process  that  will  be  familiar  to 
those  operating  in  recruitment  and  assessment. 
However,  as  shown  by  the  vertical  arrow,  what 
differentiates  this  as  a DAS  includes: 

• The  component  parts  of  the  system  are 
integrated  in  that  each  stage  generates  data  that 
feeds  into  a rolling  profile  of  the  individual 
applicant.  The  assumption  is  that  the  system  is 
constructed  to  capture  data  related  to  a clear 
competency  profile  or  library  of  such  profiles. 
Thus,  rather  than,  let  us  say,  an  application 
form  stage  and  a subsequent  CBA  stage 
operating  as  discrete  elements  with  different 
rules  and  types  of  outputs,  they  are  parts  of  an 
integrated  system  using  a common  assessment 
framework  into  which  all  data  flows. 

• That  the  cost  of  the  assessment  transactions  at 
any  stage  of  this  process  are  actively  managed 
by  a systematic  approach  that  also  adapts  the 
breadth  and  depth  of  the  assessment  dependant 
on  the  volumes  handled  at  any  one  stage. 
So,  breadth  rather  than  depth  of  the  assessment 
are  the  focus  at  the  initial  stages  where 
volumes  are  high,  while  depth  and  breadth  of 
assessment  are  introduced  at  those  stages 
where  they  become  more  cost  effective. 

• At  each  stage,  data  can  be  mined  to  deliver  a 
risk  analysis  to  support  decision  making. 
At  the  first  stage,  decisions  might  be  made 
using  a minimum  standards  approach  (in  UK 
language  a minimum  competence  approach), 
eliminating  those  falling  into  the  higher  risk 
categories.  This  select-out  focus  can  the  be 
changed  at  the  later  stages  to  a select-in 
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principle,  and  issues  such  as  placement 
(i.e.  which  role  or  career  track  should  the 
person  be  assigned  to)  can  also  be  actioned 
(indeed,  this  could  be  an  aspect  of  the  system 
built  in  at  the  earlier  stages), 

• Data  is  available  for  transmission  within  the 
organisation  so  that  geographical  spread  of 
decision-makers  is  no  longer  a problem. 
For  some  organisations  where  geography  is  not 
a problem,  it  is  the  geography  of  the  applicant 
that  is  catered  for.  For  those  organisations 
where  the  geography  of  decision-makers  is  an 
issue,  then  the  geography  of  both  applicants 
and  decision-makers  is  catered  for. 

• The  added  value  of  the  data  culled  and  mined 
through  this  process  does  not  end  when  a 
selection  decision  is  made.  When  the  person 
selected  enters  service,  then  the  data  gathered 
on  their  competencies,  their  strengths  and 
development  needs,  also  enters  with  them. 
As  such,  the  DAS  shown  in  Figure  1 is 
effectively  the  front  end  of  a Human  Resources 
Information  System  (HRIS)  which  is  populated 
with  data  by  applicants,  and  which  the 
organisation  can  then  mine  for  training  and 
development  purposes  as  well  as  making 
selection  decisions. 

The  added  value  of  DASs  to  public  relations, 

fairness  and  diversity,  and  to  extending  the  reach 


of  psychological  assessment.  The  most  obvious 
candidate  as  an  assessment  tool  for  the  first,  high 
volume  stage  of  this  system  is  biographical  data 
(biodata).  However,  the  potential  for  such  a tool  is 
greater  than  simply  capturing  data.  As  will  be 
shown  in  the  presentation,  while  e-recruitment  is 
growing  fast,  it  is  also  dominated  by  CV 
(or  resume)  machines  which  vary  in  their 
sophistication  from  electronic  forms  (i.e.  simply 
replacing  a paper  form)  to  text  search  machines. 
Yet,  neither  actually  seize  the  benefits  of  either 
automated  processing  (the  electronic  forms  have  to 
be  read  by  someone  either  on  screen  or  on  hard 
copy,  so  the  grunt  work  remains)  or  of  a valid 
psychometric  backbone  (while  resume  machines 
are  in  use  in  the  UK,  the  IPD  report  mentioned 
earlier  also  cites  the  issue  that  companies  are 
finding  the  quality  of  those  recruited  via  the 
internet  to  be  moderate  to  poor). 

The  extension  of  the  biodata  component  of  a DAS 
serves  a number  of  very  important  purposes.  First, 
research  consistently  shows  that  softer  items 
(such  as  those  tapping  competencies)  are  as,  if  not 
more,  valid  predictors  of  performance  than  harder 
items  about  qualifications,  education  etc  (see 
Stokes  et  al.,  1994,  for  examples).  So,  building  on 
the  competency  profiling  that  gives  the  DAS  its 
shape  and  purpose,  biodata  feeds  directly  into  the 
rolling  assessment  described  earlier  (indeed,  in  the 
US,  harder  biodata  items  generally  only  serve  to 
operate  as  gross  negative  disqualifies  or  GNDs  in 


Figure  1 : Model  of  a Distributed  Assessment  System  for  Selection 
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terms  of  the  absolute  must-haves  for  a job  such  as 
driving  license  or  absence  of  a criminal  record). 
Second,  research  also  shows  that  biodata  has 
minimal  adverse  impact  on  minority  applicants.  As 
such,  the  inclusion  of  competency  based  biodata 
instruments  in  the  DAS,  and  rolling  this  data  into 
other  data  collected  at  later  stages,  not  only 
improves  the  cost  management  of  recruitment 
through  the  DAS,  but  also  adds  to  the  validity  and 
fairness  of  the  final  selection  decisions. 

The  third  aspect  of  the  DAS  at  the  front  end  that 
competency  biodata  can  serve  is  that  of  public 
relations.  First,  on-line  support  can  be  provided  to 
those  coming  onto  the  recruitment  site  (the  whats 
and  whys  of  the  job,  role  or  career  track,  as  well  as 
the  how  of  the  application  process),  but  also 
feedback  on  the  outcome  of  this  first  assessment 
can  also  be  provided.  Thus,  applicants  get  a faster 
turnaround  to  their  application,  but  they  also  get 
added  value  in  a diagnostic  against  the 
competencies  for  the  job  and  not  just  a well-done 
or  goodbye  letter.  What  is  more,  assessment  serves 
as  a vehicle  for  attraction  in  an  increasingly 
competitive  employment  market. 

Much  of  the  latter  functionality  (though  not  the 
competency  based  biodata  assessment)  can  be 
found  on  the  British  Army  recruitment  site 
(one  that  has  won  many  awards  for  its  innovation). 
Another  aspect  of  that  site  also  feeds  into  one 
future  aspect  of  a DAS  that  will  serve  to  close  this 
paper.  The  British  Army  site  has  the  Army 
Challenge  that  comprises  a series  of  screens 
through  which  the  applicant  engages  in  a 
battlefield  problem.  This  serves  to  give  the  site  life 
and  to  inform  the  applicant’s  decision  to  apply  or 
not  (i.e.  an  interactive  realistic  job  preview).  It  also 
demonstrates  that  other  forms  of  assessment  can  be 
deployed  at  the  very  earliest  stages  of  the  DAS. 
In  the  presentation,  I will  describe  dynamic 
assessments  currently  being  developed  that 
incorporate  elements  of  the  situational  judgement 
(Motowildo  et  al.,  1990)  and  naturalistic  decision- 
making (Klein  et  al.,  1993)  paradigms  in 
psychological  assessment.  To  this  can  be  added  the 
possibility  of  also  including  elements  from  the 
assessment  centre  approach  at  any  stage  within  the 
DAS.  Thus,  the  capability  of  a DAS  to  provide  a 
range  of  assessments  at  any  selection  stage  is 
considerable. 

Some  concluding  comments ... 

I hope  that  this  paper  has  served  to  convey  the 
potential  of  the  DAS  concept  with  which  SHL  and 


others  are  now  actively  engaged,  and  that  DASs 
represent  a new  threshold  for  assessment  and 
selection  systems  for  military  officer.  The  driver  in 
terms  of  utility  that  was  there  in  1993  remains 
today,  and  rightly  so.  However,  the  potential  of 
what  was  being  offered  in  1993  can  now  be 
extended  within  distributed  assessment  to  an  effect 
that  was  not  imaginable  when  RSG  15  drew  up  its 
report.  Furthermore,  the  term  system  becomes  all 
the  more  powerful  with  a DAS  that  has  a valid 
psychometric  backbone  and  that  operates  as  the 
front  end  of  a HRIS.  With  tomorrow’s  world 
already  upon  us,  one  of  the  key  challenges  for 
officer  selection  must  be  the  development  of  DASs 
today. 
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